Wireless independent tracking system for use in firearm simulation training

ABSTRACT

A wireless independent tracking system (WITS) module for use with a weapon simulator in a virtual reality (VR) environment. The WITS module includes active tags for tracking a gun simulator the WITS module is coupled to; a shot detector that detects a shot being fired by the gun; a shot counter that counts a number of shots fired; a transmitter that sends a signal to a smart magazine of the gun when the gun is out of virtual ammunition; and a coupler configured to couple the WITS module to a rail of the gun.

CLAIM OF PRIORITY

This application is a CIP of U.S. patent application Ser. No. 15/865,731 entitled FIREARM SIMULATION AND TRAINING SYSTEM AND METHOD filed Jan. 9, 2018, the contents of which are incorporated by reference as if fully set forth herein.

FIELD OF THE EMBODIMENTS

Disclosed embodiments relate to firearm simulation and training, and more particularly to firearm simulation and training utilizing motion tracking.

BACKGROUND

Military training programs of the US Armed Forces are vital for mission success. Military training, aimed at the accomplishment of tasks associated with a military mission, is an important part of troop readiness. Without proper training, soldiers may be unprepared for an actual battle. Similarly, law enforcement personnel benefit from training for various situations they may encounter during their patrols. In law enforcement, personnel may not have much advance notice of when they will be in a dangerous situation. Thus, periodic training exercises are important for law enforcement to maintain a state of readiness. It is therefore desirable to have improvements in the training of military and law enforcement personnel.

SUMMARY OF THE EMBODIMENTS

In one embodiment, there is provided a wireless independent tracking system (WITS) module for use with a weapon simulator in a virtual reality (VR) environment, the WITS module comprising: at least one tag for use in tracking a weapon simulator (gun) the WITS module is coupled to; a shot detector that detects a shot being fired by the gun; a shot counter that counts a number of shots fired; a transmitter that sends a signal to a smart magazine of the gun when the gun is out of virtual ammunition; and a coupler configured to couple the WITS module to a rail of the weapon simulator.

In another embodiment, the shot detector comprises a sensor that detects a slide motion.

In embodiments, the rail is one of a front Picatinny rail of a handgun simulator and a top rail of an assault rifle simulator.

In another embodiment, a holster is provided, configured to carry the handgun simulator without interfering with the handgun simulator being tracked.

In another embodiment, the WITS module comprises a plurality of infrared active tags that blink in predefined patterns identifiable by at least one camera of a motion tracking system to distinguish tracked guns from each other.

In another embodiment, a motion tracking system tracks the active tags and triangulates a position and orientation of the gun in six degrees of freedom.

In another embodiment, the WITS module comprises an inertial measurement unit (IMU) that compensates for the loss of tracking of occluded markers.

In another embodiment, the WITS module comprises a wireless transmitter that sends a gun firing signal to a server of a scenario management system when the gun is fired, and the scenario management system propagates gun firing data to all users in a virtual reality (VR) scenario for rendering to the users as a shot fired from the gun.

In another embodiment, the WITS module comprises a battery and battery charging manager.

In another embodiment, the WITS module of comprises an accelerometer that detects movement indicating the gun is in use, and powers down the gun when it not in use, and powers up the gun when it is in use.

In another embodiment, comprising a plurality of light emitting diodes (LEDs) that emit different colors of visible light, as battery charge status indicators, wherein different color combinations represent different combinations of battery charging state and battery charge state.

In another embodiment, the shot detector detects the speed of the slide movement to distinguish between a shot release (as a fired shot), and a slower manual release. The shot counter counts a number of shots fired since a full smart magazine was inserted into the gun. When the number of shots fired equals the number of virtual bullets in the full smart magazine, the transmitter sends a signal to the smart magazine indicating it is out of bullets. The gun maintains a slide lock position when the smart magazine is empty. To reload the gun the smart magazine must be removed and re-inserted into the gun, resetting the counter to equal the number of bullets in a full smart magazine.

In another embodiment, the WITS module comprises a connector to a button mounted on the side of the gun that turns on and off a light coupled to the gun.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a system in accordance with embodiments of the present invention.

FIG. 2 is a diagram of a client device in accordance with embodiments of the present invention.

FIG. 3 is a block diagram of components in accordance with embodiments of the present invention.

FIG. 4A-4D show a shock mount in accordance with embodiments of the present invention.

FIGS. 5A, 5B, and 5C illustrate examples of a shock mount in use.

FIG. 6 illustrates a weapon simulator in accordance with embodiments of the present invention.

FIG. 7 illustrates a weapon simulator in accordance with additional embodiments of the present invention.

FIGS. 8A-8I illustrate examples of inverse kinematic solver computations.

FIG. 9A is a front view of a wearable sensor in accordance with embodiments of the present invention.

FIG. 9B is a rear view of the wearable sensor of FIG. 9A.

FIG. 9C illustrates another exemplary embodiment of wearable sensors.

FIG. 10 is a flowchart indicating process steps for embodiments of the present invention.

FIG. 11 is a flowchart indicating additional process steps for embodiments of the present invention.

FIG. 12 is an exemplary user interface in accordance with embodiments of the present invention.

FIG. 13 shows an additional embodiment of a weapon simulator.

FIG. 14 illustrates an exemplary embodiment of a wireless independent tracking system (WITS) module.

FIG. 15A shows a perspective view of an embodiment of a WITS module.

FIGS. 15B and 15C illustrate the WITS embodiment shown in FIG. 15A mounted on handgun weapon simulators.

The structure, operation, and advantages of disclosed embodiments will become further apparent upon consideration of the following description taken in conjunction with the accompanying figures (FIGs.). The figures are intended to be illustrative, not limiting. Certain elements in some of the figures may be omitted, or illustrated not-to-scale, for illustrative clarity.

DETAILED DESCRIPTION

Disclosed embodiments provide systems and methods for conducting a firearm usage simulation including simulated firearm discharge, enabling enhanced training of armed forces and/or law enforcement personnel. A motion tracking system tracks motion of one or more users. In embodiments, the users wear one or more sensors, transponders, or other wearable devices on their bodies to allow tracking by the motion tracking system. A scenario management system utilizes position, orientation, and motion information provided by the motion tracking system to evaluate user performance during a scenario. A weapon simulator includes sensors that indicate position of the weapon and/or orientation of the weapon. The weapon simulator may further provide trigger activation indications to the scenario management system. In embodiments, the scenario management system generates, plays, reviews, and/or evaluates simulations. The evaluation can include scoring based on reaction times, posture, body position, body orientation, and/or other attributes. Thus, disclosed embodiments provide a safe and cost-effective way to train personnel in the use of a weapon in various scenarios, and evaluate the users based on information provided by the motion tracking system and/or weapon simulator.

FIG. 1 is a diagram of a system 100 in accordance with embodiments of the present invention. System 100 includes a scenario management system 102. The scenario management system 102 can be implemented in an electronic computing device that includes a processor 140, a memory 142 coupled to the processor, and storage 144, also coupled to the processor 140. Memory 142 contains instructions 147, that when executed by the processor 140, cause the scenario management system 102 to perform steps in accordance with embodiments of the present invention. Memory 142 may be a non-transitory computer readable medium. Memory 142 may include, but is not limited to, flash, read-only memory (ROM), static random-access memory (SRAM), magnetic storage, optical storage, or other suitable storage mechanism. Storage 144 may include one or more magnetic hard disk drives (HDD), solid state disk drives (SSD), optical storage devices, tape drives, and/or other suitable storage devices.

In embodiments, storage 144 may include multiple hard disk drives configured in a RAID (redundant array of independent disks) configuration. In embodiments, the RAID configuration can include a RAID 1 configuration in which data is copied seamlessly and simultaneously, from one disk to another, creating a replica, or mirror. If one hard disk drive becomes inoperable, another hard disk drive continues to operate, providing a level of fault tolerance.

The processor 140 may include multiple cores. In embodiments, the scenario management system 102 may include multiple processors, where each processor includes multiple cores. Embodiments may schedule tasks and threads over multiple processors and/or cores to achieve a level of parallelism in execution of various tasks such as computations, rendering, and/or scenario generation.

Embodiments may include additional redundancy through failover mechanisms. In embodiments, the scenario management system 102 may be instantiated as a virtual machine operating in a cloud environment. In embodiments, multiple instantiations of the scenario management system 102 may be implemented in the cloud environment. Scenario management system 102 is in communication with network 124. Network 124 may be the Internet, a local area network (LAN), wide area network (WAN), or another suitable network.

The term “Internet” as used herein refers to a network of networks which uses certain protocols, such as the TCP/IP protocol, and possibly other protocols such as the hypertext transfer protocol (HTTP) for hypertext markup language (HTML) documents that make up the World Wide Web (web). The physical connections of the Internet and the protocols and communication procedures of the Internet are well known to those of skill in the art. Access to the Internet can be provided by Internet service providers (ISP). Users on client systems, such as client 116 obtains access to the Internet through the Internet service providers. Access to the Internet allows users of the client computer systems to exchange information, receive and send e-mails, and view documents, such as documents which have been prepared in the HTML format. These documents are often provided by web servers which are considered to be “on” the Internet. Often these web servers are provided by the ISPs, although a computer system can be set up and connected to the Internet without that system being also an ISP as is well known in the art.

System 100 further includes motion tracking system 130. Motion tracking system 130 includes a processor 132, a memory 134 coupled to the processor, and storage 136, also coupled to the processor 132. Memory 134 contains instructions, that when executed by the processor 132, cause the motion tracking system 130 to perform steps in accordance with embodiments of the present invention. Memory 134 may be a non-transitory computer readable medium. Memory 134 may include, but is not limited to, flash, read-only memory (ROM), static random-access memory (SRAM), magnetic storage, optical storage, or other suitable storage mechanism. Storage 136 may include one or more magnetic hard disk drives (HDD), solid state disk drives (SSD), optical storage devices, tape drives, and/or other suitable storage devices. In embodiments, memory 134 includes at least 32 gigabytes of RAM. Motion tracking system 130 further includes one or more cameras 137. The cameras may be configured to detect visible and/or infrared light. The motion tracking system 130 may further include one or more sensors. These sensors may include, but are not limited to, temperature sensors, proximity sensors, noise sensors, and/or other suitable sensors.

In some embodiments, processor 132 may include an Intel i7 CPU or higher. The motion tracking system 130 may include a graphics processor such as the Nvidia GTX 1070 or higher, and include a high-performance network card.

A client device, indicated as 116 may be connected to network 124 via a wired or wireless interface. Client device 116 may include a mobile computing device such as a smartphone or tablet, a laptop computer, a desktop computer, or other suitable computing device. The client-server architecture allows a user to remotely access features of the scenario management system 102. In embodiments, the client device 116 may include an Intel i7 CPU or higher, an Nvidia GTX 1080 graphics processing unit or higher, and 16 GB of ram or more.

Embodiments of the present invention may utilize a JavaScript Object Notation (JSON) web service to make a JSON call to the scenario management system. In some examples, the JSON call is made using XML HTTP, which implements an XML HTTP object that has functionality enabling the exchange of Extensible Markup Language (XML) data directly over the Internet using the Hypertext Transfer Protocol (HTTP). The XML HTTP object allows access of the scenario management system data from a server, parsing the data using an XML Document Object Model (DOM), and posting XML data through a standard firewall directly to an HTTP server.

The cameras 137 and/or sensors 138 of motion tracking system 130 may be deployed in a venue such as a room, building, or outdoor area, such that they can track the motion of one or more users (200A, 200B, 200C, and 200D). Note that while four users are shown in FIG. 1, in practice, more or fewer users may be present. Referring now to user 200A, additional detail is shown. Each user may utilize one or more wearable sensors 206. The wearable sensors can be used to detect motion and position of a user. By having sensors on the limbs of a user, the position and/or orientation of a user can be more precisely determined. Each user may further wear a helmet 205. The helmet 205 may include a sensor 207 that can be used to determine head location and/or head orientation of the wearer (user). Each user may further wear goggles 204.

The goggles 204 may include virtual reality goggles, augmented reality goggles, or other suitable eyewear. The goggles 204 may further include speakers that provide audible feedback to a user. In embodiments, the goggles 204 may include a forward-facing camera to enable recording and/or monitoring of the user's point of view. In embodiments, the goggles 204 may include a user-facing camera to monitor the eye gaze of a user. In this way, the direction of eye gaze, even if different from head orientation, can be ascertained.

Each user may further utilize a weapon simulator 202. The weapon simulator may be in the form of a firearm, and may include a recoil simulation mechanism such as compressed air or a mechanical spring to simulate the recoil associated with discharging a real firearm. The weapon simulator 202 may further include an inertial tracker 208 affixed to the weapon simulator 202. The inertial tracker 208 may include one or more accelerometers and/or gyroscopes to track motion of the weapon simulator 202. Sometimes, motion associated with recoil can adversely affect the inertial tracker 208. Therefore, embodiments may further include a shock mount 210 disposed between the weapon simulator 202 and the inertial tracker 208. This allows the position of the weapon simulator to be tracked as the user moves it, while preventing the adverse effects of recoil motion, since the shock mount absorbs some of the recoil motion.

FIG. 2 is a block diagram of a client device 300 in accordance with embodiments of the present invention. In embodiments, client device 300 is an electronic device that may include a desktop computer, laptop computer, tablet computer, smartphone, and/or other suitable client device. Client device 300 may be similar to client device 116 as shown in FIG. 1. Client device 300 includes a processor 302, a memory 304 coupled to the processor 302, and storage 306. The memory 304 may be a non-transitory computer readable medium. Memory 304 may include RAM, ROM, flash, EEPROM, or other suitable storage technology. The memory 304 contains instructions, that when executed by processor 302, enable communication to/from scenario management system 102 of FIG. 1. Client device 300 further includes a network communication interface 310 for performing this communication. In embodiments, network communication interface 310 includes a wireless communications interface such as a cellular data interface and/or a Wi-Fi interface. In embodiments, the storage 306 includes flash, SRAM, one or more hard disk drives (HDDs) and/or solid state disk drives (SDDs).

Device 300 may further include a user interface 308. User interface 308 may include a keyboard, monitor, mouse, and/or touchscreen, and provides a user with the ability to enter information as necessary to utilize embodiments of the present invention. In embodiments, a user uses the device 300 to access the scenario management system 102.

FIG. 3 is a block diagram 400 of components in accordance with embodiments of the present invention. Components include, but are not limited to, motion tracker 402, weapons interface 404, inverse kinematic solver 406, client communication component 408, scenario management component 410, and/or data analysis 412. The motion tracker 402 may be a commercially available motion tracker such as OptiTrack™ by NatrualPoint, of Corvallis, Oreg. The motion tracker 402 may include multiple cameras installed within a room, building, or other venue where a scenario is to be executed. The cameras may be adapted to track wearable sensors that emit infrared light. As a user, wearing the wearable sensors moves, his/her motion is tracked b the motion tracker 402. The weapons interface 404 may implement recoil simulation, gun flash simulation, trigger activation notifications, and/or other weapons-related simulation functions. The inverse kinematic (IK) solver 406 may be used to create an inverse kinematic solution to rotate and position links in a chain. This can be used to model human movement to create virtual targets and/or model the motion of live users.

The scenario management component 410 may be implemented as a computer system such as system 102 of FIG. 1. Alternatively, the scenario management component 410 may be implemented as multiple computers in a distributed or cloud computing environment. The scenario management component may be used for generating scenarios, executing scenarios, and/or playback of previously executed scenarios. The data analysis component 412 may be used for analyzing an executed scenario. This can include performing numerous assessments on users that are tracked by the motion tracker 402. Assessments can include reaction time, location analysis, physiological orientation of the user, orientation of the user's head, eye gaze, limb position, and others.

For example, in a given scenario, upon hearing a gunshot, a user may be trained to drop to a crouched position, turn his head towards the direction of the gunshots, and draw his weapon. To provide a training experience for a user, a scenario is executed by the scenario management component 410. This component may generate the virtual environment, including audiovisual information rendered by goggles 204. A gunshot sound is rendered on the speaker of goggles 204 worn by the user, and the scenario management component 410 records this time. The user starts to move, and the motion tracker determines how long it takes to get into a crouched position, how long it takes for the user to draw his weapon, and if his head is oriented in the proper position. In embodiments, after a predetermined time (e.g. 2 seconds), the user orientation and weapon orientation are evaluated. In embodiments, a user score is generated based on the time required for the user to achieve the proper position. In this way, the user can continually practice, and review his performance to achieve an optimal reaction time. Other, more complex examples are possible, such as scenarios utilizing multiple friends and foes. In embodiments, one or more of the friends and/or foes may be virtual.

FIG. 4A-4D show a shock mount in accordance with embodiments of the present invention.

FIG. 4A is a perspective view of a shock mount 500 in accordance with embodiments of the present invention. Shock mount 500 includes a top plate 503, a bottom plate 506, and a plurality of resilient members 504 configured and disposed between the top plate 503 and the bottom plate 506, and terminated at the top with end caps, indicated generally as 541. This reduces the amount of shock imparted to the inertial tracker 508 from the handle 526 of a weapon simulator as a result of a recoil simulation. The weapon simulator may utilize compressed air, solenoids, springs, or other electromechanical mechanisms to simulate a recoil that occurs when a firearm is discharged. The shock mount 500 reduces the shock imparted on the inertial tracker 508, which helps improve the reliability of data returned by the inertial tracker. In embodiments, the inertial tracker may be a commercially available inertial tracker such as Vive™ tracker by HTC corporation. In some embodiments, the resiliency of the resilient members 504 may be adjustable to accommodate different types of weapon simulators and/or inertial trackers. Shock mount 500 may be similar to shock mount 210 shown in FIG. 1. FIG. 4B is a top view of the shock mount 500, illustrating the mounting shoe 543 configured and disposed to receive a weapon simulator. FIG. 4C is a front view of the shock mount 500, illustrating a plurality of coupling pins 545 for interfacing to an inertial tracker. In embodiments, the coupling pins 545 may be spring loaded “Pogo” pins. FIG. 4D is a bottom view of the shock mount 500, indicating a linear arrangement of the coupling pins 545.

FIG. 5A illustrates an example of a shock mount in use. A weapon simulator 520 has a handle 526. In embodiments, the weapon simulator 520 may be in the form factor of a pistol. The shock mount 500 is affixed to the bottom of the handle (grip) 526. The inertial tracker 508 is affixed to the shock mount 500. The weapon simulator 520 shown is a Glock simulator. In embodiments, the weapon simulator 520 is configured to use compressed gas in order to create haptic feedback (or recoil) that creates a strong vibration to the inertial tracker 508 tracker. The shock mount 500 of disclosed embodiments connects the inertial tracker 508 to the handle 526 while also reducing the shock from simulated recoil which could possibly ruin the tracking data of the inertial tracker 508. In embodiments, a small pressure-sensitive button is placed behind the trigger to send the trigger activation information through electronic circuitry that connects to an interface of the inertial tracker. The inertial tracker may then use a wireless interface such as Bluetooth™ to send the trigger activation signal to the computer for further processing.

FIGS. 5B and 5C illustrate more examples of a shock mount in use.

FIG. 6 illustrates a weapon simulator 600 in accordance with embodiments of the present invention. Weapon simulator 600 is in the form factor of an automatic rifle. A motor 602 embedded in the grip is configured to generate haptic feedback upon activation of the trigger 604. A battery 612 in the stock of the weapon simulator 600 provides power for the motor 602. Additionally, a wireless transmitter 606 is configured and disposed to provide discharge information to the scenario management system. The wireless transmitter 606 may communicate with a wireless receiver 608 attached to a computer 610. In embodiments, computer 610 is part of the scenario management system. Thus, embodiments include a communication module configured and disposed to wirelessly transmit a trigger indication to the scenario management system in response to activation of the trigger.

FIG. 7 illustrates a weapon simulator 700 in accordance with additional embodiments of the present invention. Weapon simulator may be in the form factor of a pistol. An optical tracker 702 may be installed on the weapon simulator 700. One or more light sources, indicated as 704 and 706 may be tracked by the motion tracking system to provide information on the position and/or orientation of the weapon simulator to the scenario management system.

In embodiments, the tracker 702 orients one or more LEDs in specific locations for optimal tracking. A pressure sensitive button is placed behind the handgun trigger and that passes a signal to a computer via a wireless communication protocol such as Bluetooth™, WiFi, Zigbee, or other suitable protocol upon activation of the trigger 708 of the weapon simulator. In embodiments, the unit is self-contained with its own LIPO battery, voltage converter, charger port, and on/off button.

FIG. 8A-8F illustrate examples of inverse kinematic solver computations. In embodiments, a full body IK solver allows animation of a human body model in real time based on animations of 5 control points. In embodiments, a body skeleton is divided into the following five bone chains. A spine chain starts with a pelvis bone and ends with a head bone. A left leg chain starts with a pelvis bone and ends with a left foot bone. A right leg chain starts with a pelvis bone and ends with a right foot bone. A left arm chain starts with a left shoulder bone and ends with a left palm bone. A right arm chain starts with a right shoulder bone and ends with a right palm bone. Each chain of bones is solved using a Backward/Forward IK solver gets as an input, two 3D matrices and calculates transform of bones in a chain in 3D space. Changing locations or rotations of either start or end input will change transformation of the bones in the chain. Each chain can consist of any number of bones. Referring now to FIG. 8A, a bone chain 800 is shown in an extended bone state. Three bones (802, 803, and 804) are extended fully such that endpoint 801 and endpoint 805 are at a maximum possible distance.

Referring now to FIGS. 8B, 8C, and 8D, there are shown examples of non-extended bone states. Referring to FIG. 8B, bone chain 810 is oriented such that the distance D between first endpoint (start) 811 and second endpoint (end) 812 is less than the sum of the lengths of all bones in the bone chain. A similar case exists for bone chain 814 of FIG. 8C and bone chain 815 of FIG. 8D.

Referring now to FIG. 8E there is a bone chain with a force vector UP applied to it. In order to bend the chain in the known direction, the UP direction has to be defined for the chain. UP direction is a 3D vector. UP direction gets applied to the initial pose and changes it. In the diagram, the UP direction is to the left, which means the chain 819 will bend to the right.

Referring now to FIG. 8F, forward solving computations are shown. Starting from the current state of the chain, positions of each bone in the chain are found using the forward solving equation: A′=End′−(End′−A)

Starting from the bone closest to the end, this equation is repeated for each bone by replacing

End′ vector with the location of previous bone in the chain.

For finding B′ for example, the following equation is used: B′=A′−(A′−B)

Similarly, for C′: C′=B′−(B′−C)

Referring now to FIG. 8G. a chain rigidity factor is applied. In embodiments, each bone has a rigidity factor which determines how easily a given bone will change its position when start or end transforms change. A value from 0 to 1 determines where the bone will be placed between the initial pose and the current pose. In this example, bone 821 has a rigidity factor of 0.5 and so it is moved to halfway between points B and B′.

Referring now to FIG. 8H, backward solving is performed. Starting from the current state of the chain, positions of each bone in the chain are found using a backward solving equation. The process starts from the bone closest to the start input, and the equations below are applied: C″=C B″=C″+(B′−C″) A″=B″+(A′−B″)

Referring now to FIG. 8i , the orientation of the bones is calculated. Bone chain 823 shows an initial position, and bone chain 825 shows a final position. In embodiments, orientation of the bones gets calculated after positions are solved for, and it is known where each bone is located and where this bone was in the initial pose. In embodiments, in the initial pose a rotational matrix is constructed for each bone. In the solved pose there is constructed a new rotational matrix for each bone using the same axis order. One axis is parallel to the length of the bone (Z in the diagram). A second axis is linearly interpolated between the upnode (UP) chain vector multiplied by start and upnode chain vector multiplied by end transform (Y in the diagram). Linear interpolation of this axis allows simulation of twisting of bones along their long axis (z) if end or start transform is twisted. This approach allows adjusting of the extent of the twist for each bone easily by changing one numeric value. Once there are derived two rotational matrices for the bone, then the offset between them is calculated and this transform offset is applied to the bone. This approach allows rotation of bones independently of their initial orientation. Orientation of bones can be inconsistent in the initial chain pose.

FIG. 9A is a front view of a wearable sensor 900 in accordance with embodiments of the present invention. Wearable sensor 900 includes light emitting diode 902 installed therein. A battery within the sensor (not shown) applied power to the light emitting diode 902. In embodiments, the light emitting diode (LED) 902 emits infrared light and does not emit substantial visible light. This allows the LED 902 of sensor 900 to be detected by the motion tracking system, and yet not be noticeable to the users.

FIG. 9B is a rear view of the wearable sensor 900 of FIG. 9A. In this view, the adjustable strap 904 can be seen. This strap 904 is used to secure the wearable sensor around a limb (arm, leg) of a user. In embodiments, the strap 904 may have a fastener such as a buckle, snap, hook and loop fastener, or other suitable fastening mechanism. Wearable sensor 900 may be similar to sensors 206 shown in FIG. 1.

FIG. 9C is another exemplary embodiment of a wearable sensor. In the figure, adjustable strap 904 is fitted to the wearer using buckles 906. One or more sensors may be contained in module enclosure 910, which is coupled to the strap 904 by a coupling mechanism that releases the enclosure by pressing release tab 908. The illustrated embodiment includes tracker elements 912, which may be active or passive. The tracker module 910 may comprise one or more sensors, transceivers, processors, and the like.

FIG. 10 is a flowchart 1000 indicating process steps for embodiments of the present invention. At process step 1002, a user position is determined. The user may be wearing one or more wearable devices such as position sensors and/or position indicators. The wearable devices may emit infrared light that is tracked by a motion tracking system. In process step 1004, an initial physiological orientation is determined. In some embodiments, a wearable device is worn on each limb, allowing the motion tracking system to determine physiological orientation. The physiological orientation can include a stance (e.g. standing, crouching, prone), and may further including a direction the user is facing. In process step 1006, a simulated firearm discharge is detected. In embodiments, this can include a weapon simulator in the form factor of a firearm. The weapon simulator includes a power source (battery) and a wireless transmitter that is configured and disposed to transmit a trigger activation indication to a computer to indicate that the weapon simulator has discharged (e.g. a user pulled the trigger on the weapon simulator). The weapon simulator may or may not actually shoot a projectile. In process step 1008, a user reaction is determined. This can include determining a second physiological orientation of the user at some predetermined time after the detection of the simulated firearm discharge. The second physiological orientation can include a position of a person, position of the limbs of that person, orientation of the torso of the person (which way the torso is facing), and/or orientation of the head of the person (which way the head of the user is facing). Thus, in embodiments, determining a physiological orientation of the user comprises determining a facial direction of the user.

Additionally, in embodiments with a user-facing camera (e.g. included in virtual or augmented reality goggles), an eye gaze direction may further be included in the physiological orientation. Various attributes can then be evaluated, including, but not limited to, the time required for the user to achieve the second physiological orientation (e.g. a measure of how long did it take the user to get into a crouch position in response to hearing and/or seeing a simulated firearm discharge), the correctness of the second physiological orientation as compared to a known orientation (e.g. a measure of if the user's crouch is sufficiently low), and/or the time required to draw a weapon (e.g. a measure of the time required for the user to pull a weapon simulator from a holster and orient it in a position to fire). Other attributes of user reaction may be evaluated instead of, or in addition to the aforementioned attributes in embodiments of the present invention.

FIG. 11 is a flowchart 1100 indicating additional process steps for embodiments of the present invention. In process step 1102, a final position time is determined. This may include determining a time where the user's level of motion is below a predetermined threshold. Thus, embodiments include further comprising measuring a duration from a time of the simulated firearm discharge to a time of detecting the determined physiological orientation. As an example, when a user is in the process of quickly moving from a standing position to a crouched position, there is a relatively large amount of motion. Once the user is stabilized in the crouched position, there is relatively less motion. The time when the user is substantially still may be recorded as a final position time. At process step 1104, a final position quality is determined. This may include utilization of motion tracking data to determine how close the user's physiological orientation is to an expected physiological orientation. This can include an assessment of if the user is in a proper stance/position, facing the proper direction, appropriately drawing his weapon simulator, and/or other factors. In process step 1106, the final position time is compared to average times. In process step 1108, a score is generated based on the comparison to the average time. In some embodiments, the score may be generated using the formula: K(1−X) Where: K is a constant; X is the final position time in seconds.

As an example, if an average time for a law enforcement professional to go from standing to crouched is 400 milliseconds, then the final position time may be compared against the average time using the above formula. If K is 116, then the following score is achieved for different crouch times:

Crouch time (milliseconds) Score 400 69.6 500 50 600 46.4 200 92.8 300 81.2

As can be seen from the table above, the formula returns a score such that a user with an average time of 400 milliseconds earns a score of about 70, while a user with a score of 300 returns a higher score (81.2). Similarly, a user with a relatively slow time of 600 milliseconds returns a low score of 46.4. This concept can also be applied to other attributes, such as position quality, weapon draw time, weapon discharge time, and/or other attributes. In embodiments, the score for each attribute may be combined into an overall score.

FIG. 12 is an exemplary user interface 1200 in accordance with embodiments of the present invention. User interface 1200 may be implemented on a client device such as client device 116 of FIG. 1. The user interface may be used to allow users and/or trainers and coaches to review performance. User interface 1200 includes a video window 1202 that may show video and/or a virtual animated rendition of the user performance. In some embodiments, real video and animated graphics may be composited together in video window 1202. User interface 1200 further includes a name field 1204, a scenario name 1206, a final position score 1208, a reaction time 1210, a weapon draw time 1214, and a weapon discharge time 1216. In embodiments, the weapon draw time 1216 may be the time required by the user to move his weapon simulator from a holstered position to a ready position as determined by the motion tracking system. In embodiments, the weapon discharge time may be the time required by the user to fire the weapon simulator after getting the weapon simulator into the ready position (e.g. aimed at a target). The data analysis component (412 of FIG. 3) may further provide computer-generated recommendations, which may be rendered in field 1218. The recommendations can be based on scores and/or times for one or more attributes during a scenario execution. The user interface 1200 is merely exemplary, and other user interfaces are possible showing more, fewer, or different fields in some embodiments.

FIG. 13 shows an additional embodiment. Weapon simulator 1300 includes additional stock 1302 affixed to the rear end of the weapon simulator. This enables users wearing a head-mounted display (HMD) to get into a shooting position without having the HMD bump into the stock end of the weapon simulator, thereby allowing for a more unencumbered user experience.

Some embodiments may provide an untethered experience with computers mounted in backpacks. Such embodiments may include, but are not limited to, an MSR V1 with a GTX 1070, or an HP ZVR with a GTX 1070. The computers may be configured with an Intel i7 processor and at least 16 GB of RAM. Other backpack computers are possible in embodiments of the present invention.

As can now be appreciated, disclosed embodiments provide an improved training system for firearm usage. A motion tracking system tracks the motion of one or more users. Users use a weapon simulator that integrates with a scenario management system. The scenario management system thus obtains information about weapon position and weapon discharge, as well as position of the users. The scenario management system can generate scenarios where live participants, virtual participants, and/or computer-generated targets work together or against each other to conduct training drills. Through the use of virtual reality and/or augmented reality, various landscapes, terrain, buildings, and other factors can be simulated. The reactions of users can be timed and assessed, allowing for improved review of the performance of users such as military and law enforcement personnel. In this way, the effectiveness and safety of these people can be continuously monitored and improved.

Referring now to FIG. 14, embodiments may include a Wireless Independent Tracking System (WITS) module 1400. A WITS module is a stand-alone unit for use in a VR environment in conjunction with airsoft weapons, simunition weapons, real weapons with blanks, and the like (hereinafter, guns). The WITS module provides for tracking the gun and detecting when a shot has been fired. The WITS module may further send a signal to a smart magazine of the weapon when it's out of VR bullets, and cause the gun to mimic a slide-lock action. As shown in the figure, for handguns the WITS module 1400 may be mounted on a front Picatinny rail or other location. A holster (not shown) may be provided with the handgun, configured to enable tracking the handgun while it is in the holster. On an assault rifle weapon simulator, the WITS module 1400 may be mounted on a top rail, right above the bolt mechanism, or other location. In other embodiments, the weapon simulator may be designed in the form of a rifle, a shotgun, a taser, or the like. Depending on the design of the gun and the configuration of the WITS module, the WITS module may be provided with a coupler 1435 for attaching the WITS module to the gun. The coupler may be or include one or more hook and loop fasters, buttons, snaps, straps, buckles, bolts, or the like, or combinations of these.

In embodiments, the WITS module may have a case 1405 enclosing other components of the module. WITS module components comprise tags 1455, which may be commercially available active tags. Illustratively, Optitrack's infrared active tags may be used in training scenarios in conjunction with a control board. The board may cause the active tags 1455 to blink in distinct patterns that may be identified by the motion tracking system's cameras and associated with particular guns. That is, by reading the active tags' blinking patterns, the system can distinguish a plurality of guns from each other and identity and track them all individually. In embodiments, the system may track the active tags 1455, and from their position may triangulate the location, position, and/or orientation of the guns in 6 degrees of freedom (forward and back, left and right, up and down).

In embodiments, one or more of the active tags 1455 may be associated with an inertial measurement unit (IMU) 1410 that compensates for the loss of line of sight tracking when one or more markers 1455 are occluded by objects in the training environment. In embodiments, a commercially available IMU 1410 may be used. Illistratively, an IMU developed by Optitrack may be used. Further, such IMUs may be found in commercially available training systems such as systems available from V-Armed (https://www.v-armed.com), which uses this technology in their tracking devices.

The WITS module 1400 contains sensors and wireless communication components that may be implemented using a custom printed circuit board (PCB) comprising onboard components. When a gun equipped with the WITS module is fired, a WITS shot detector 1415 containing one or more sensors may detect the gun being fired, such as by detecting the trigger pull, or a slide movement, or a recoil or other firing indicator. In response, the WITS module sends a signal to a computer such as a server of a scenario management system using a wireless transmitter 1430 of any appropriate communications system, such as an IEEE 802.11(x) compliant WiFi system, or a long range (LoRa) physical layer-compliant system, or the like. The response data, or a derivative thereof generated by the scenario management system, is then propagated to all participating trainees in the VR scenario. The propagated information may include audio information, visual information, or both, which may be rendered to the trainees as a VR shot from the fired weapon. The shot may be rendered to a trainee via a head or helmet mounted apparatus including a speaker, virtual display, or both, for example. The WITS module may further be used in conjunction with a gun magazine reloader, as will be described.

In embodiments, the WITS module 1400 may include an accelerometer 1450 that automatically detects when the weapon simulator is idle, and puts the weapon simulator to sleep to save battery life. In such an embodiment, when the accelerometer detects motion it immediately causes the weapon simulator to energize and be ready for action.

The WITS module may contain its own battery 1440 and battery charging manager 1445. In embodiments, a charging station (not shown) may be configured and arranged to charge the gun with mounted WITS module. Colored LEDs 1460 may be included on the WITS module to indicate charging progress when a gun with mounted WITS module is coupled to the charging station. For example in an exemplary embodiment, RGBY LEDs 1460 may indicate charging progress as follows:

-   -   RED battery charge low     -   GREEN battery charge ok     -   BLUE Connected to Wifi     -   YELLOW battery charging

Combinations of LED colors may also be used, such as:

-   -   YELLOW/RED Charging, battery charge low     -   YELLOW/GREEN Charging, battery charge ok     -   BLUE/RED Connected to Wifi, battery low     -   BLUE/RED/YELLOW Connected to Wifi, battery low and charging     -   BLUE/GREEN/YELLOW Connected to Wifi, battery charge ok and         charging.

Other colors and color combinations may additionally or alternatively be used.

In embodiments, a gun may include a smart magazine (“smart mag”, not shown). The smart mag may be configured as a modified airsoft magazine that works in collaboration with the WITS module. In embodiments, the WITS shot detector 1415 detects changes in the speed of the slide movement to determine when a shot is fired. In these embodiments, a shot release slide movement may be used to detect when a shot is fired. In contrast, a manual slide movement is always slower than a shot release slide movement, so the WITS module is able to distinguish between these two actions.

When a fully loaded smart mag is inserted into the gun, a counter 1420 of the WITS module is set to the number of bullets the smart mag contains. When the shot detector 1415 detects a shot, the counter 1420 subtracts one from the number of bullets remaining in the smart mag, until the number of bullets left is zero, indicating the smart mag is empty. An empty signal transmitter 1425 then sends a signal to the smart mag indicating it has no more bullets. In order to reload, the user presses a gun release button 1465 to remove the smart mag from the gun. The user then re-inserts the smart mag into the gun and presses a slide release button 1470, similar to the operation of a real gun. At that point the gun is ready to fire again.

When the smart mag has no bullets left, it may enable the gun to be put into a slide-lock position similar to a real gun when the user is out of bullets. However, without the smart mag, the gun may continue to cycle when using the WITS module, even if it is out of virtual bullets. In this situation, the gun will not actually fire in a VR scenario, even though it may continue to give haptic feedback or the like during training. Although this situation is unrealistic, for the most part it may not adversely affect the VR experience because emptying a fully loaded magazine in many scenarios is unusual. Then, to virtually reload the weapon simulator without a smart mag, the user may simply pull the slide back manually to remove a virtual mag from the gun, and a new virtual mag will appear in the gun, which is then ready to fire again.

In embodiments, the WITS module may have a quick connection 1475 for an extra button or switch, that can be mounted on the side of the gun to turn on or off a real or virtual light (not shown) that may be coupled to the gun.

FIG. 15A is a perspective view of one embodiment of a WITS module 1500. As shown, the coupler includes a mounting bracket 1535, shown disposed on the top of the module case 1505, ready to be mounted to a bottom rail of a gun. The module could also be flipped so that the mounting bracket 1535 is disposed on the bottom of the module for mounting on a top rail of a gun. FIGS. 15B and 15C illustrate a WITS module 1500 mounted under the barrel of handgun weapon simulators.

At least part of the present invention may be embodied in a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a non-transitory computer readable storage medium or media, having computer readable program instructions stored therein for causing a processor to carry out aspects of the present invention.

Each of the above methods may be executed on one or more processors on one or more computer systems. Embodiments may include various forms of distributed computing, client/server computing, and cloud based computing. Further, it will be understood that the depicted steps or boxes contained in the disclosed flowcharts are solely illustrative and explanatory. The steps may be modified, omitted, repeated, or re-ordered without departing from the scope of this disclosure. Further, each step may contain one or more sub-steps. While the foregoing drawings and description set forth functional aspects of the disclosed systems, no particular implementation or arrangement of software and/or hardware should be inferred from these descriptions unless explicitly stated or otherwise clear from the context. All such arrangements of software and/or hardware are intended to fall within the scope of this disclosure.

The block diagrams and flowchart illustrations depict methods, apparatus, systems, and computer program products. Any and all such functions, generally referred to herein as a “circuit,” “module,” or “system” may be implemented by computer program instructions, by special-purpose hardware-based computer systems, by combinations of special purpose hardware and computer instructions, by combinations of general purpose hardware and computer instructions, and so on.

It will be understood that a computer may include a computer program product from a computer-readable storage medium and that this medium may be internal or external, removable and replaceable, or fixed. In addition, a computer may include a Basic Input/Output System (BIOS), firmware, an operating system, a database, or the like that may include, interface with, or support the software and hardware described herein.

Embodiments of the present invention are neither limited to conventional computer applications nor the programmable apparatus that run them. To illustrate: the embodiments of the presently claimed invention could include an optical computer, quantum computer, analog computer, or the like. A computer program may be loaded onto a computer to produce a particular machine that may perform any and all of the depicted functions. This particular machine provides a means for carrying out any and all of the depicted functions.

Any combination of one or more computer readable media may be utilized including but not limited to: a non-transitory computer readable medium for storage; an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor computer readable storage medium or any suitable combination of the foregoing; a portable computer diskette; a hard disk; a random access memory (RAM); a read-only memory (ROM), an erasable programmable read-only memory (EPROM, Flash, MRAM, FeRAM, or phase change memory); an optical fiber; a portable compact disc; an optical storage device; a magnetic storage device; or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device. Program data may also be received via the network adapter or network interface.

These computer readable program instructions may be provided to a processor of a computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

It will be appreciated that computer program instructions may include computer executable code. A variety of languages for expressing computer program instructions may include without limitation C, C++, Java, JavaScript™, assembly language, Perl, Python, Ruby, hardware description languages, database programming languages, functional programming languages, imperative programming languages, and so on. In embodiments, computer program instructions may be stored, compiled, or interpreted to run on a computer, a programmable data processing apparatus, a heterogeneous combination of processors or processor architectures, and so on. Without limitation, embodiments of the present invention may take the form of web-based computer software, which includes client/server software, software-as-a-service, peer-to-peer software, or the like.

In embodiments, a computer may enable execution of computer program instructions including multiple programs or threads. The multiple programs or threads may be processed approximately simultaneously to enhance utilization of the processor and to facilitate substantially simultaneous functions. By way of implementation, any and all methods, program codes, program instructions, and the like described herein may be implemented in one or more threads which may in turn spawn other threads, which may themselves have priorities associated with them. In some embodiments, a computer may process these threads based on priority or other order.

Unless explicitly stated or otherwise clear from the context, the verbs “execute” and “process” may be used interchangeably to indicate execute, process, interpret, compile, assemble, link, load, or a combination of the foregoing. Therefore, embodiments that execute or process computer program instructions, computer-executable code, or the like may act upon the instructions or code in any and all of the ways described. Furthermore, the method steps shown are intended to include any suitable method of causing one or more parties or entities to perform the steps.

The terminology used herein is for describing particular aspects only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “include” and “including” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Certain examples and elements described in the present specification, including in the claims and as illustrated in the figures, may be distinguished or otherwise identified from others by unique adjectives (e.g. a “first” element distinguished from another “second” or “third” of a plurality of elements, a “primary” distinguished from a “secondary” one or “another” item, etc.) Such identifying adjectives are generally used to reduce confusion or uncertainty, and are not to be construed to limit the claims to any specific illustrated element or embodiment, or to imply any precedence, ordering or ranking of any claim elements, limitations or process steps.

While the invention has been disclosed in connection with preferred embodiments shown and described in detail, various modifications and improvements thereon will become apparent to those skilled in the art. Accordingly, the forgoing examples should not limit the spirit and scope of the present invention; rather it should be understood in the broadest sense allowable by law. 

What is claimed is:
 1. A wireless independent tracking system (WITS) module for use with a weapon simulator in a virtual reality (VR) environment, the WITS module comprising: at least one tag for use in tracking a weapon simulator (gun) the WITS module is coupled to; a shot detector that detects a shot being fired by the gun; a shot counter that counts a number of shots fired; a transmitter that sends a signal to a smart magazine of the gun when the gun is out of virtual ammunition; and a coupler configured to couple the WITS module to a rail of the weapon simulator.
 2. The WITS module of claim 1, wherein the shot detector comprises a sensor that detects a slide motion.
 3. The WITS module of claim 1, wherein the rail is one of a front Picatinny rail of a handgun simulator and a top rail of an assault rifle simulator.
 4. The WITS module of claim 3, wherein the WITS module is provided with a holster configured to carry the handgun simulator without interfering with the handgun simulator being tracked.
 5. The WITS module of claim 1, wherein the tags include a plurality of infrared active tags that blink in predefined patterns identifiable by at least one camera of a motion tracking system to distinguish tracked guns from each other.
 6. The WITS module of claim 5, wherein a motion tracking system tracks the active tags and triangulates their position, and determines from the tags' positions an orientation of the gun in six degrees of freedom.
 7. The WITS module of claim 1, further comprising an inertial measurement unit (IMU) that compensates for loss of tracking of the markers when they are occluded.
 8. The WITS module of claim 1, further comprising: a wireless transmitter that sends a gun firing signal to a server of a scenario management system every time the gun is fired; wherein the scenario management system then propagates gun firing data to all users in a virtual reality (VR) scenario for rendering to the users as a shot fired from the gun.
 9. The WITS module of claim 1, further comprising a battery and battery charging manager.
 10. The WITS module of claim 1, further comprising an accelerometer that detects when the weapon simulator is in use, wherein the WITS module powers down the gun when it detects the gun is not in use, and powers up the gun when it determines the gun is in use.
 11. The WITS module of claim 1, further comprising a plurality of light emitting diodes (LEDs) that emit different colors of visible light, as battery charge status indicators; wherein different color combinations represent different combinations of battery charging state and battery charge state.
 12. The WITS module of claim 1, wherein: the shot detector detects a speed of a slide movement to distinguish between a shot release slide movement (as a fired shot), and a manual slide movement that is slower than the shot release slide movement; and the shot counter counts a number of shots fired since a full smart magazine was inserted into the gun; and when the number of shots fired equals the number of virtual bullets in the full magazine, the transmitter sends the signal to the smart magazine indicating the smart magazine is empty; wherein the gun is in a slide lock position when the smart magazine is empty, and to reload the gun: a magazine release button must be pressed to remove the empty smart magazine, the smart magazine must be re-inserted (as a full magazine) into the gun, and a slide release button must be pressed, resetting the counter to equal the number of bullets in a full smart magazine.
 13. The WITS module of claim 1, further comprising: a connector to a button mounted on the side of the weapon simulator that turns on and off a real or virtual light coupled to the weapon simulator. 